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Climate change literacy plays a key role in promoting sound political decisions and 
promoting sustainable consumption patterns. Based on evidence suggesting that student 
understanding and interest in climate change is best accomplished through studying 
local effects, we developed a simulation/game exploring the impact of climate change on 
the declining water levels in Lake Mead. Because there are few evaluations of 
educational games using true control groups, this study also presents a randomized field 
trial evaluating the game. We randomly assigned 119 seventh graders to either a game- 
based condition or control condition. Students in the experimental group played Losing 
the Lake; those in the control group viewed an earth science website. Students also 
completed pretest, posttest, and delayed posttest measures of their content 
understanding and interest in issues embodied in the game. We found that playing the 
game resulted in a significant increase in content knowledge, as measured by a 22-item 
assessment, especially on items related to household conservation and some basic 
concepts related to the greenhouse effect. The control group showed no effect. Playing 


Correspondence: E. Michael Nussbaum, 

Department of Educational Psychology and Higher Education, University of Nevada, 
Las Vegas, 4505 S. Maryland Parkway, Box 453003, Las Vegas, NV, USA 
E-mail: nussbaum(5)unlv.nevada.edu 
doi: 10.12973/ijese.2015.277a 


Copyright © 2015 by iSER, International Society of Educational Research 
ISSN:1306-3065 






E. M. Nussbaum et al. 


the game also resulted in some increase in student interest. The Losing the Lake game 
illustrates how use of a water theme can be used to make climate change content more 
meaningful and relevant to students. Furthermore, the study shows, through a 
randomized control trial, that educational games can result in conceptual development, 
specifically on water flow (i.e., where drinking water comes from and where it goes once 
used), water conservation, and the difference between weather and climate. The Losing 
the Lake game can therefore be useful educationally in various locales as a case study in 
the nature of drought, climate change effects, and water conservation practices. 

Keywords-, simulations, sustainability, climate change education, water resource 
management, educational game 

INTRODUCTION 

Anthropogenic climate change has caused a variety of adaptation and mitigation 
challenges for modern societies. These include increased frequency of drought, 
floods, storms, and other natural disasters, as well as consequences related to 
disease (Intergovernmental Panel on Climate Change [IPCC, 2007]). Climate change 
literacy plays a key role in promoting sound political decisions and promoting 
sustainable consumption patterns (Anderson, 2012). However, few K-12 teachers 
have received any formal training on climate change and cite a lack of knowledge, as 
well as the complexity of the topic, as barriers to providing quality climate change 
education (National Research Council, 2012; Wise, 2010). Understanding climate 
change requires systems thinking (Mirchi, Madani, Watkins, & Ahmad, 2012; 
Sterman et al., 2012), including an understanding of various natural systems 
(carbon cycle, water cycle, etc.) and social systems (economic, political, etc.). 
Systems thinking involves understanding interaction and feedback effects within 
these systems (Shepardson, Roychoudhury, Hirsch, Niyogi, & Top, 2014). Finally, 
helping students understand climate change requires addressing and overcoming 
various misconceptions, such as confusing weather with climate or climate change 
with ozone depletion (Papadimitriou, 2004). It also involves promoting critical 
thinking about climate-related issues. 

It is therefore important for teachers and students to be provided with 
instructional resources that enhance student understanding of the topic, for 
example through computer-based educational environments (Songer, 2007). The 
use of multimedia materials can transform the way students learn through visual 
engagement and promoting active learning (Plass et al., 2012). Computer 
simulations and games, if properly designed, have also been shown to promote 
systems thinking and understanding (Blikstein & Wilensky, 2009). Another body of 
research has shown that effective climate change education needs to focus on 
content that is tangible, meaningful, and actionable (Anderson, 2012). One such 
content area is water resources. Water is life-sustaining and something that all 
humans need and can relate to. Climate change, by altering long-term precipitation 
patterns, will exacerbate droughts, flooding, and other water-related phenomena 
(Carrier, Kalra, & Ahmad, 2013; Forsee & Ahmad, 2011; Kalra & Ahmad, 2012). 

Although the number of computer-based climate change serious games has 
increased over the last few years (Reckien & Eisenack, 2013), few of these games 
focus specifically on water, and many of the latter were designed for water 
managers and decision makers (Valkering, Van der Brugge, Offermans, Haasnoot, & 
Vreugdenhil, 2012), not K-12 students. We therefore sought to develop an 
educational game that we call Losing the Lake. The game explores the effect of 
climate change on water resources in the southwestern U.S., specifically on water 
levels in Lake Mead. Lake Mead is the largest human-created reservoir in the United 
States and a major source of drinking water for three Western states and Mexico. 
The lake is fed by the Colorado River (Sagarika, Kalra, & Ahmad, 2014), and the river 
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is fed by water from the Rocky Mountain snowpack. An extended drought over the 
last 14 years, made worse by climate change, has reduced the snowpack in the 
Rocky Mountains and, in turn, Lake Mead water levels.. The steadily declining water 
levels in the lake has had consequences which are highly visible, as shown in Figure 

1 . 



Figure 1 . Lake Mead water level as of 2010. The white "bathtub” ring reflects exposed rock 
that was previously underwater. The ring shown here is approximately 130 feet (40 m] 
(©Photo by Lynn F. Fenstermaker, used with permission.] 

Images such as these can be powerful symbols of the effects of climate change. 
Imagery is useful pedagogically for attracting student attention and illustrating local 
relevance (Messaris, 1997; O'Neill & Nicholson-Cole, 2009], We designed the game 
to further support student understanding of two complex systems/phenomena: the 
water supply system in Southern Nevada and the global greenhouse effect, as well as 
water conservation efforts. Although the design of the game has been described in 
some detail elsewhere (Nussbaum, Sinatra, Cordova, Owens, & Rehmat 2012; 
Nussbaum, Sinatra, & Owens, 2012; Vesco et al., 2012], this article briefly describes 
the game and—unlike previous reports—presents an experimental study of the 
game's effectiveness. The simulation models that provided some data for the 
activities on water conservation and lake levels are described in Dawadi and Ahmad 
(2012, 2013], We report on specific content areas where the game did (or did not] 
increase student knowledge, as well as effects on student interest and motivation. 

Conceptual framework 

The game (and study] address aspects of climate change literacy. Anderson 
(2012] presents several design principles of effective climate change education 
based on a systemic literature review conducted by the Brookings Institution, in 
conjunction with graduate students at Columbia University and New York 
University. The principles are as follows (from Anderson, 2012, pp. 198-199], 

1. "Climate change literacy can be improved through sustained, active learning 
activities using integrated, cross-discipline curricula.” 
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2. “Active learning should be connected to local problems solving.... Educational 
techniques that connect climate change with not only local issues but also 
individual behavior and impacts offer tremendous promise, both by making 
abstract concepts tangible and by linking the global phenomenon to 
individual action" (Duan & Fortner, 2005], 

3. "While climate change education should inform students about the scientific 
concepts and implications of climate change, it is also important to cultivate 
problem solving and critical thinking skills through framing messages to 
emphasize an individual’s capacity to achieve positive outcomes" (Morton, 
Rabinovich, Marshall, & Bretschneider, 2011; Spence & Pidgeon, 2010], 
Research has shown that appeals to fear and negative consequences can 
cause disengagement (O'Neil & Nicholson-Cole, 2009], whereas appeal to 
things that individuals can do to adapt to or mitigate negative consequences 
builds self-efficacy and stimulates deeper processing (Moser & Diller, 2007], 
Climate change education therefore needs to be action oriented (National 
Research Council, 2011], 

4. "Narrative techniques, visual imagery (such as photographs] and persuasive 
texts are powerful tools” that make global climate change dynamics more 
understandable and engaging, and impact students’ attitudes and behavioral 
intentions" (Niepold, Herring, & McConville, 2008; Sinatra, Kardash, 
Taasoobshirazi, & Lombardi, 2011], 

In view of these design principles, we searched for a local problem related to 
climate change that was visually rich and that would invite students to problem 
solve. A local problem that met these criteria was the declining water levels in Lake 
Mead. The Nevada System of Higher Education had been awarded a $15 million 
grant from the National Science Foundation to research local problems related to 
climate change and strengthen infrastructure, outreach and education efforts. 
Through an interdisciplinary science award from the larger grant, our team of 
educators partnered with a civil and environmental engineer (Dr. Sajjad Ahmad] 
and two computer scientists at the University of Nevada, Reno (Dr. Fred C. Harris, Jr., 
& Dr. Sergiu M. Dascalu] to develop the Losing the Lake computer game. The game 
was intended as an instructional resource for middle school students as well as for 
the general public in informal environments (for example, the game was placed in 
the Las Vegas Natural History Museum], The game was a freestanding resource in 
the latter but in the middle school environment could be combined with other 
instructional activities. We developed a facilitator's manual containing suggestions 
for activities and discussions that could be combined with the game. Both the game 
and the manual are available on the web at: 

http://sensor.nevada.edu/NCCP/Education/Losing%20the%20Lake/Default.asp 

x. 

However, in this study, our goal was to evaluate the effects of just the game on 
student learning and motivation. Although we hypothesize that the effectiveness 
would be greater if combined with other instructional activities, testing this 
assumption is not our present goal. From a theoretical perspective, this study was 
meant to shed light not only on the effectiveness of Losing the Lake but also more 
broadly on the utility of using a water theme in connecting the meaningfulness of 
the issue of climate change to student learning and interest. 

Game design 

The 'Losing the Lake' game was designed with five modules/activities to help 
break down the complex topic of climate change. This specific design was employed 
with the idea that the game could be used as a whole or as modules for teaching the 
concepts of climate change and water conservation. Additionally, the 
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modules/activities were created so the students could easily comprehend and digest 
climate change content and not be overwhelmed by the game as a whole. Table 1 
describes the five modules/activities of the game. 

Table 1 . Description of game activities 

Activity Purpose Description 


Multiple- 

choice 

questions (6) 


Building background knowledge 
regarding snowfall, drought, 
climate change, greenhouse gases. 


If users answer incorrectly, they can try answering again, and 
if still incorrect, are provided written feedback. Three 
alternative question sets were developed in the event that a 


user wants to replay the activity or game. 


Household Building knowledge of 

conservation conservation activities; enhancing 

self-efficacy. 


Users select from a menu of options (e.g., fix leaky toilets, 
cover swimming pool, etc.) the three that will save a 
household the most water. There is a visual of the inside of a 
house and parts of it glow (e.g., toilet, swimming pool) when 
the related option is selected. Users earn points for their 
performance. 


Community Building knowledge of 

conservation conservation activities; enhancing 

self-efficacy. 


Same as Activity 2 but options relate to things local or state 
policymakers could do to conserve water (e.g., raise water 
prices, promote desert landscaping). Some are poor options 
(e.g., turn off resort fountains). Activity performed against 
map of Las Vegas. 


Climate 

change 


Build a mental model regarding Consists of a series of vocal explanations, visuals, and small 

why lake levels are declining and simulations regarding temperature, snowpack levels, 
connect it to climate change. greenhouse gas emissions, and lake levels. 


Simulation 
of lake water 
levels 


Apply previous concepts to a 
context like Lake Mead; illustrate 
how scientific models are used. 


Users must predict how long it will be before lake levels 
decline to 1,000 feet (305 meter) above sea level. The 
simulation is then run on a visual model of the lake so that 
users can see the water declining. The computer simulations 
are based on hydrologic models using the Hadley Center 2- 
A1B global climate model data. Users earn points for how 
closely their prediction matches the model. 
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Table 2. Menus of potential conservation options in activities 2 and 3 


Activity 2 (Household) 


Activity 3 (Community) 


Fix leaking toilets & faucets 
Replace dishwasher 
Replace washing machine 
Cover swimming pool 
Remove lawns 
Do not wash car at home 
Plant trees 

Bathe instead of shower 
Don't water lawn if windy or rainy 
Install low flow faucets 
Replace showerheads 
Bathe or shower less often 


Restrict resort water usage 
Turn off resort fountains 
Reduce golf course watering 
Newer homes use grey water 
Water smart landscaping 
Pool regulation 
Raise water prices 
Control population growth 


OCL- 
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Average water loss Average water loss 
without your changes with your changes 


You Chose: 

Replace dishwasher 

This is an OK option! 

An energy-efficient dishwasher not only 



This is not a good option! 

Baths use more water than showers. It 
takes more water to fill your bathtub than 
to take a 5 minute shower. 


saves you more energy and water than your 
old dishwasher, but it also saves more water 
than hand-washing your dishes. You could 
save up to 10 gallons of water per load. But 
most people don’t run their dishwashers 
every day. 

Remove lawns 

This is a great option! 

Every square foot of grass that you remove 
saves up to 55 gallons of water per year. If 
you remove 300 square feet of lawn, you 
can save up to 16,500 gallons of water per 
year! 


Bathe Instead of shower 


LM.irain s 


iis i 



a] Example of feedback screen on household conservation options (Activity 2] 
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Compared to current 
conditions, your selection 
would keep "Lake Heed" 
froni;falling below 1,000 feet 
for an additonal 5 years. 


© 


LM l[ffl 


(b] Example of one simulation result from Activity 5. 


Figure 2. Screenshots of Losing the Lake. 


In designing the game, we were cognizant of design principle #3, regarding 
building students' self-efficacy for action and avoiding fear appeals. On the other 
hand, some level of alarm was needed to reflect the reality of climate change impacts 
and attract and hold student attention to the effects needed for mitigation. As a 
compromise, we designed the game (specifically Activity 2] to first highlight the 
problem (that a major source of drinking water is declining] but then the game 
quickly turns to the issue of water conservation in students’ household (choose the 
three options from a menu that would save the most water] and then in Activity 3, in 
the community. (These options are shown in Table 2.] Students receive informative 
feedback and game points depending on the options selected (the amount of water 
saved is illustrated by water flowing into a water cooler, with sound effects. See 
Figure 2a], An early focus on conservation in Losing the Lake, specifically starting in 
Activities 2 and 3, is intended to alleviate anxiety and build self-efficacy. 

Losing the Lake is also designed so that users focus on increasingly more complex 
environments and systems. The first activity is a series of six questions intended to 
build students' background knowledge (see Table 3], As noted above, the second 
activity focuses on household conservation, and the focus is expanded in the third 
activity to community conservation. The fourth activity is an extended tutorial (with 
mini-activities for which users earn points] that links the problem to climate change. 
The activity is cognitively complex because the user has to understand all of the 
following: 

• Most of the drinking water in the Las Vegas Valley comes from Lake 
Mead. 

• Precipitation in the form of snow creates the snowpack in the Rocky 
Mountains. 

• Melting snow from the Rocky Mountain snowpack feeds the Colorado 
River, which in turn feeds Lake Mead. (The geography is shown to 
students.] 

• Snow levels have been steadily declining, as have lake levels. 
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• Some of this decline is due to climate change (which refers to changes 
over a 30-year period or longer], 

• Climate change is linked to the "enhanced" greenhouse effect, which will 
cause the planet to be hotter and rain and snow fall to decline in the 
region. (Users are asked to guess by what percent. The link between 
greenhouse gas emissions and global temperature is also reinforced with 
a brief graphical simulation involving a thermometer.] 

• An increase in global temperatures will cause regional average 
temperatures to increase (users predict the extent], 

• An increase in regional temperatures will reduce precipitation and 
reduce snow levels in the region. (Users simulate this relationship— 
based on actual forecasts—by moving a slider representing temperature 
that reduces a representation of snow levels.] 

• The game ends with a slide explaining that "the increase in temperature 
that you just saw is an average for the whole year. It means there will 
probably be less snowfall in the winter, earlier spring warming, more 
water evaporation, and therefore less water in the Colorado River for us 
to use" (in the long-run]. 

Table 3. Examples of background knowledge questions 

The most important factor in determining the amount of water flowing in the Colorado River is the 

amount of_. 

• Rainfall in the area 

• Snowfall in the Rocky Mountains 

• Snowfall in the Sierra Nevada Mountains 

• Water flowing from other rivers into the Colorado 


Greenhouse gas emissions contribute to global warming. In the United States, where do most of the 
greenhouse gas emissions come from? 

• Waste management processes 

• Volcanic gases 

• Agriculture 

• Burning of fossil fuels 


Although the game would be less complex if students were not asked to make 
estimations, making such predictions followed by experimentation or simulation is a 
widespread practice in science education. It promotes active cognitive processing 
and conceptual change (through cognitive conflict] when there is a discrepancy 
between predicted and actual results (Kearney, Treagus, Yeo, & Zadnik, 2001], 

Whereas the fourth activity shows how climate change will make the on-going 
drought worse in the decades ahead, the fifth activity returns to the issue of 
conservation, but from a regional perspective. Conservation in just the Las Vegas 
Valley will have little effect on Lake Mead water levels because Nevada uses only a 
small allocation from the lake (its allocation, less than 2%, is the smallest amount of 
the seven states with a water allocation]; however, regional conservation can make 
a difference. Users are introduced to the concept of mathematical and hydrologic 
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modeling, and then are asked to pretend that they are water resource managers for 
a fictional lake in the southwestern U.S. (known as "Lake Heed"] which is like Lake 
Mead and has water levels that are dangerously low. They make predictions 
regarding which of four options (20% municipal water conservation, doubling 
municipal water prices, 20% agricultural water conservation in California, and 
reducing global greenhouse gas emissions] will save the most water and extend the 
number of years before lake levels fall below 1,000 feet (305 meters] above sea 
level. (When the level drops below 1,000 feet, no more water can be drawn from the 
lake for Las Vegas because of the location of the intake pipes.] User predictions are 
then compared against actual results from scientific modeling using the global 
climate model Hadley Center 2-A1B data (with points earned based on how closely 
user predictions match the simulated ones]. The results of one simulation are 
shown in Figure 2b. 

Because there are a number of other global climate models, and because existing 
conditions change (e.g., "current" water levels in Lake Mead], we did not want the 
simulation results from our game taken out of context, quoted by the media, or used 
inappropriately; the results are illustrative for pedagogical purposes only. This was 
the rationale for using a fictional lake in Activity 5. 

At the end of the game, users are told that "we need to do a combination of all 
these options" and are then given their overall point total. High scores are 
reinforced with the adjectives "good" or "great." The last screen thanks the user for 
playing and directs them to go save some water. Users can also select some options 
for more information (links to other websites]. 

Playing the game takes approximately 20 minutes. The game has verbal 
narration, but the text of the narration is also shown on the screen in the event that 
audio is not available for some reason (e.g., noise levels are too high]. 

The current study evaluated the effectiveness of the game by examining whether 
there would be an impact on measures of both student knowledge and interest. 

METHODS 

Participants 

To evaluate the effectiveness of the game, we recruited two 7 th -grade earth 
science teachers to work with us on the evaluation; the teachers were from two 
different middle schools in a large, urban school district in the southwestern U.S. 
Both teachers taught seventh grade and allowed us to conduct the study in all their 
class sections (six sections for the teacher at School A and five sections for the other 
teacher at School B], When the study commenced, the teacher at School B had 
already completed a unit on weather and climate and the other teacher was just 
beginning a similar unit. The unit addressed the characteristics of the atmosphere 
and its effects on Earth’s surface, energy transfer in the atmosphere and oceans, 
greenhouse gases, solar energy and the water cycle (Holt Science & Technology, 
2006], 

A total of 119 students participated in the study; of these, 21 students were 
Caucasian, 17 Asian, 66 Hispanic, 8 African American, and 7 other. 51% were males. 
Schoolwide, 19.9% were limited-English proficient at School A and 42.4% at School 
B. 

Materials 

Materials included a 22-item quiz developed by the research team to assess the 
content knowledge taught in the game. The items were primarily multiple-choice 
and true/false selected response (see appendix for the specific items], but two 
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constructed-response items were also included, one pertaining to the importance of 
water conservation in the Las Vegas Valley and the other to water conservation 
options. For the first item ("Why is it important for the Las Vegas Valley specifically 
to conserve water?"], responses were scored on a 0-3 scale blind to condition. 
Responses received a score of 1 for mentioning a connection to Lake Mead, a score 
of 2 for mentioning lake water levels, and a score of 3 for mentioning drought or 
living in a desert. Fifty percent of the responses were scored by a second rater, and 
the inter-rater reliability was 86%. For the second item ("Describe three good ways 
of saving water in your home or apartment,”] respondents received the same 
number of points for mentioning a specific conservation option as were awarded in 
the game. Table 4 shows the specific point values for the different options. (See 
Nussbaum, Sinatra, Cordova et al., 2012, for the rationales for the point values 
shown in the table.] 


Table 4. Point values for different household conservation options 


Option 

Points 

Fix leaky toilets and faucets 

2 

Replace dishwasher 

0 

Replace washing machine 

4 

Cover pool 

7 

Remove lawns 

7 

Do not wash car at home 

1 

Plants trees 

0 

Bathe instead of shower 

0 

Install low flow faucets 

6 

Replace showerheads 

5 

Bathe or shower less often 

0 

Don't water when windy or rainy 

7 


We also developed a 10-item interest survey to assess students' interest in 
learning about water conservation and climate change and a student feedback 
survey about the Losing the Lake game (see Tables 5 and 6], The items used 4-point 
Likert scales ranging from 1 ("no interest"] to 4 ("high interest"]. As described 
below, the survey was administered both before and after the intervention. The 
internal consistency of the interest survey was .81 before and .86 after the 
intervention. The feedback survey (which was administered just to the 
experimental group after viewing the game] consisted of seven 6-point Likert-scale 
items ranging from 1 (Strongly Disagree] to 6 (Strongly Agree], The internal 
consistency of these items was .89. The feedback survey also contained four open- 
ended items regarding what students learned, liked, and disliked about the game, 
and regarding what they would like to learn more about after playing Losing the 
Lake. 

Finally, we prepared written sheets with a weblink to the game (for the 
experimental groups] and to a general earth science website (for the control group]. 
The control group link was: < www.eduweb.com/portfolio/earthsvstems >. The site 
contained short sections on different Earth Systems but contained no information on 
climate change. The specific topics covered included the Earth’s crust and 
atmosphere, food webs, hydrology, and glaciers. The researchers monitored 
students when viewing the website and made sure that students remained on task. 
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Table 5. Student interest survey items 


1. Knowing more about where the Las Vegas Valley gets its water. 

2. Learning more about water conservation in the home. 

3. Knowing more about water conservation in the community. 

4. Learning more about the Colorado River system. 

5. Learning more about climate change and global warming. 

6. Understanding better how human activity impacts climate change. 

7. Knowing more about how climate change affects water availability. 

8. Understanding better how humans impact water availability. 

9. Knowing more about scientific modeling. 

10. Understanding how scientific modeling is used in making predictions and guiding decision making. 


Notes. Items rated on a 4-point Likert scale (1 = no interest, 2 = low interest, 3 = medium interest, & 4 = high 
interest). Instructions to students were: "Please rate your degree of interest in the following activities by circling a 
number from 1 (No interest) to 4 (High Interest). Your answers will be kept strictly confidential and will not be 
identified by name.” 

Table 6. Student feedback survey items 


1. 1 enjoyed playing Losing the Lake. 

2. I was bored playing Losing the Lake. 

3. I want to play Losing the Lake again. 

4. Playing Losing the Lake seemed to drag on forever. 

5. Losing the Lake was interesting to me. 

6. I liked playing Losing the Lake. 

7. I think playing Losing the Lake is a waste of time. 

8. Please list at least two things you liked about Losing the Lake. 

9. Please list at least two things you didn't like about Losing the Lake. 

10. Please list at least two things you want to learn more about after playing Losing the Lake. 

Notes. Items 1-7 rated on a 6-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = somewhat disagree, 4 = 
somewhat agree, 5 = agree, 6 = strongly agree). 

Design and procedures 

We used a pretest/posttest (and delayed posttest] randomized two-group 
experimental design, using the following procedures. First, parental consent and 
student assent forms were collected during the first few weeks of the semester. The 
actual study was split into four sessions. During the first session (25 min in 
duration], participating students completed a short demographics form, the pretest, 
and an interest survey. During the second session, which took place a day or two 
later, students were randomly divided—within each class—into experimental and 
control groups. (Assignments were made the day before using a random number 
generator in Excel.] Students in the experimental group went to the school's 
computer lab and played "Losing the Lake" for 25 min while the other students 
worked on unrelated schoolwork (e.g., reading their textbooks]. Then students 
switched, with the control group going to the computer lab, where they explored the 
earth science website while the experimental group participants completed 
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unrelated schoolwork. Session three occurred the following day and lasted 25 min; 
all participants completed the posttest, the interest survey and student feedback 
survey. Lastly, during the fourth and final session, which occurred a week or two 
later, all participants were given 25 min to complete the delayed posttest. 

Scoring 

For each applicable testing session, participants received scores for: (a] the 
objective items on the quiz, (b] each of the two constructed response items, (c] the 
interest survey, and (d] the student feedback survey (Likert items]. One item 
(multiple choice #8] was removed from the objective test analysis because of poor 
discrimination (.03 posttest/delayed posttest] and ambiguity. The student feedback 
survey constructed responses were coded for emerging themes using 
nonhierarchical coding (Johnson & Christensen, 2004], with frequency counts 
tabulated for different categories. 

Data analysis 


Table 7. Means (& SDs, n] of outcome measures 


_ Measure _ 

Objective test (prej 
Object test (post] 

Objective test (delayed post] 
Open-ended item 1 (pre] 
Open-ended item 1 (post] 
Open-ended item 1 (delayed post] 
Open-ended item 2 (pre] 
Open-ended item 2 (post] 
Open-ended item 2 (delayed post] 
Interest(pre] 

Interest (post] 


Condition 


Experimental 

Control 

7.12 

6.95 

(1.97,59] 

(1.70,61] 

8.36 

6.83 

(2.47, 59] 

(1.77, 60] 

8.20 

6.86 

(2.80, 54] 

(1.89, 58] 

0.78 

1.31 

(1.22,63] 

(1.36, 62] 

0.97 

1.35 

(1.31,63] 

(1.34, 62] 

0.95 

1.17 

(1.35,63] 

(1.42, 62] 

1.13 

1.41 

(2.68,60] 

(3.26,61] 

5.35 

2.62 

(6.29,60] 

(4.47, 60] 

6.63 

4.05 

(6.65,51] 

(5.08, 58] 

22.21 

23.01 

(4.32,61] 

(4.26, 60] 

23.10 

22.96 

(4.70, 59] 

(5.29, 60] 


For the statistical analyses, we used robust standard-error estimators and the 
generalized linear model (GLM] function in SPSS version 19 to conduct the analysis. 
(Because of repeated measures over time, Generalized Estimating Equations were 
used, with an autoregressive-1 error structure.] The specific GLMs fitted were: (a] 
for the objective test, normal errors and a quadratic functional form (given that we 
did not expect a linear trend over time], (b] for the first open-ended item, a 
cumulative logistic model (given the ordinal nature of the scoring], and (c] for the 
second open-ended item, a gamma error structure with the log-link (given that the 
data were heavily skewed toward zero], (All scores were increased by 1 point to 
prevent scores of zero, which do not have logarithms.] For the analysis of the 
interest items, a standard repeated measures ANOVA was conducted as no 
assumptions of the general linear model were violated. 
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For each test, we performed five planned comparisons. We hypothesized that the 
experimental group would show an increase from pretest to posttest and from 
pretest to delayed posttest, and that the means for the experimental group would be 
higher than for the control group at posttest and delayed posttest but not at pretest. 
The Bonferroni-adjusted a = .01. The means and SDs for each outcome measure are 
shown in Table 7. 

RESULTS 

Objective test scores 

As we had hypothesized, the objective test scores significantly increased in the 
experimental group from pretest to posttest/delayed posttest but there was no gain 
in the control group. The omnibus test was significant (Wald x 2 = 9.83, p = .002] for 
the interaction between group and time (see Table 8 for the complete test of model 
effects]. 


Table 8. Objective test model effects 


Term 

B 

95% Cl 

Wald x 2 

p-value 

(Intercept] 

4.43 

[2.55, 6.30] 

105.95*** 

<.000 

Group (G] 

-3.33 

[1.01, 5.65] 

y g 

.005 

Time (T) 

3.37 

[1.24, 5.50] 

3.41 

.065 

T 2 

-0.71 | 

[-1.22,-0.19] 

2.43 

.119 

GxT 

4.24 | 

[-6.89,-1.59] 

9.83** 

.002 

GxT 2 

-0.89 

[0.23, 1.54] 

7.06** 

.008 

Notes. Model fit: QICC 

= 1.478.214; Scale estimate 4.33. 



*p < .05. **p < .01. ***p < .001. 




Table 9. Objective test expected marginal means 

by time and group 



Expected Means 


Time 

Experimental 

Control 

(Total] 

Pretest 

7.09 


7.07 

7.08 

Posttest 

8.35 


6.75 

7.55 

Delayed Post 

8.19 


6.79 

7.49 

(Total] 

(7.88] 


(6.87] 

(7.37] 


Wald 95% Confidence Interval 


Time 

Experimental 

Control 

Total 

Pretest 

[6.60, 7.58] 


[6.68, 7.46] 

[6.77, 7.39] 

Posttest 

[7.72, 8.97] 


[6.34, 7.15] 

[7.18, 7.92] 

Delayed Posttest 

[7.47, 8.91] 


[6.31, 7.28] 

[7.06, 7.93] 

Total 

[7.38, 8.38] 


[6.56, 7.18] 

[7.08, 7.67] 
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The expected marginal means are shown in Table 9. The scores for the control 
group remained flat, but those for the experimental group increased from pretest to 
posttest. The increase from pretest to posttest was significant (p < .001, d = 0.63] as 
was the increase from pretest to delayed posttest (p = .001, d = 0.56], In addition, 
the experimental group held their gain showing only a nonsignificant decline at the 
delayed posttest (p = .87], There was no significant difference between the groups 
at pretest (p = .95], but there was at posttest (p < .001, d = 1.01] and delayed posttest 
(p =.002, d = 0.75], 


Table 10. Mean gain scores by item (for experimental group], from pretest to posttest 


Item stem (answer paraphrased ] 

What has the greatest effect on water levels in the 
Colorado River? ( snowpack ) 

Climate change and changes in weather are the same, [false ) 
Humans cannot survive on Earth without the greenhouse 
effect, [true] 

There are ways to conserve water at home. Some ways save more 
water than others. Which of the following is the best way to 
conserve water? [grey water ) 

What happens to the water that we are finished using in our homes? 
[returned to lake) 

There are ways to conserve water within your community. Some 

ways save more water than others. Which of the following is 
the best way to conserve water within your community? 
[raise water prices) 

Which of the following contributes to an increase in greenhouse 
gases? [car emissions) 

Which of the following is a greenhouse gas? (CO 2 ) 

Over the past 100 years global temperatures have increased, [true) 
Resorts and casinos waste a large amount of water, [false) 

Which of the following options would save the most water? 

[desert landscaping) 

Most of the water from Lake Mead is used for [agriculture) 

If Lake Mead water levels decline below 1,000 feet above sea 
level, what will then happen? [unable to pump water 
from lake) 

If climate change causes an increase in average temperature of 

the area, what happens to the water in Lake Mead? [lowers) 
Which of the following determines how much water Nevada can use 
from Lake Mead? [compact) 

Having cities cut back 20% on their water use is estimated to 

delay Lake Mead water levels from declining below 1,000 
feet (above sea levels) for how many years? [5years) 

The greenhouse effect and the ozone hole are basically the 
same thing, [false) 

Human activities produce greenhouse gases, [true) 

Grey water systems recycle water that comes from: [bathing) 
Requiring everyone in the community to cover their swimming 
pool would save very little water, [false ) 

*p<. 05. **p<. 01. ***p < .001. 


Gain %Correct 

_ (Post) 


0 49 *** 69.5 

0.19* 55.9 

0.19** 69.5 

0.19* 27.1 

0.15* 32.2 

0.12 18.6 

0.10 40.7 

0.07 50.8 

0.07 91.5 

0.05 20.3 

0.02 16.9 

0.02 8.5 

0.00 33.9 

-0.02 57.6 

0.03 32.2 

-0.05 39.0 

-0.05 67.8 

-0.05 69.5 

-0.07 16.9 

-0.14** 50.8 


Table 10 shows the specific items on which learning gains were the greatest. The 
largest (and statistically most significant] gains were on five items. Two of the items 
related to the determinants of water flow: Colorado River water volumes are 
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affected by snowpack, and water, once used by the Las Vegas Valley is returned to 
Lake Mead. Two related to climate: Climate is different from weather, and the 
greenhouse effect is needed for human survival. The final item related to household 
options for saving water. The control group also showed a gain on this item (p 
=.006], but the gain (0.13] was only half that of the experimental group. All the 
gains in the experimental group were maintained at the delayed posttest except for 
the item relating to needing the greenhouse effect for survival. The mean for the 
item relating to car emissions was also significantly greater than at pretest (0.25 
gain, p = .004], Finally, there was a significant decline on the swimming pool item 
(decline from pretest, -0.20, p =.01]. 

Performance on two conservation items was suboptimal. One question asked 
students to select which, of four options, would save the most water. The modal 
response concerned replacing old washing machines (50.8% chosen at posttest], 
whereas the correct answer was replacing lawns with desert landscaping. Likewise, 
on a community conservation item, the modal response was requiring everyone in 
the community to cover their swimming pools (19.6%], whereas the correct 
response was raising water prices. Nevertheless, pool covers was still rated in the 
game as an excellent option. For both these items, we suspect that because users 
selected and received feedback on different options, their learning about the 
different options was uneven. Our analysis of the constructed response items 
(presented below] suggests that students did learn from the game about important 
conservation options. 

Disappointingly, there was no gain on the item that asked, "If climate change 
causes an increase in average temperature of the area, what happens to the water in 
Lake Mead?" Although 57.6% of the experimental group answered the question 
correctly at posttest, a sizable minority (43%] chose the alternative "The water in 
Lake Mead will rise due to the melting of the snowpack and therefore, more water 
flowing into the Lake." In fact, this is only true in the short-term; earlier spring 
melting will increase the amount of water evaporation and, over the long-run, this 
coupled with decreased snowfall will reduce lake levels once much of the permanent 
snowpack is gone. 

In conclusion, the game did result in learning gains on the broad concepts 
addressed by the game. However, there were three popular misconceptions that 
were not effectively addressed, relating to casinos and resorts wasting a lot of water 
(untrue], the necessity of the greenhouse effect for human survival (effect significant 
at posttest but not maintained at delayed posttest], and that melting snowpack will 
cause lake levels to rise. Supplementary instruction is needed to reinforce these 
aspects of the game. 

Constructed response scores 

For the first constructed response question ("Why is it important for the Las 
Vegas Valley specifically to conserve water?"], there were no significant effects (e.g., 
for the interaction, Wald x 2 = 0.70, p = .40], The overall mean score was 1.17 on a 0 
to 3 scale. 

The second constructed response question was, "Describe three good ways to 
conserve water in your home or apartment." There was a significant effect of time 
(Wald x 2 = 22.15, p < .001, showing a linear but not quadratic trend] and significant 
Group by Time interaction (Wald x 2 = 5.40, p = .02], favoring the experimental 
group. 

Most of the planned comparisons were as hypothesized. For the experimental 
group, scores significantly increased from pretest to posttest (p < .001, d = 1.57] and 
from pretest to delayed posttest (p < .001, d = 2.05], and were significantly different 
from the control group at posttest (p = .006, d = 0.64] but not at pretest (p = .56], 
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The difference between the groups at the delayed posttest was not quite significant 
at the .01 level (p = .03, d = 0.56], This occurred because of a slight increase in 
scores in the control group. (This increase could have been due to the experimental 
group students talking about what they learned from the game with the control 
group during the delay.] 

A content analysis of the responses in the experimental groups (posttest] showed 
that the most frequently mentioned option was covering swimming pools (22.7%], 
followed by "do not wash car at home" (13.3%], "remove lawns" (12%], and "newer 
homes use grey water" (10.7%]. All these options were rated as great ones in the 
game, except for not washing cars at home, which was rated as just OK (because cars 
are typically washed only every few weeks]. The one great option that was 
infrequently mentioned (2.7%] was "Don't water if windy or rainy"—therefore, 
additional supplementary instruction may be needed on this item. 

Interest survey 

Items 9 and 10 were deleted from the analysis because there were a very high 
number of missing values. (These items were on the back of the survey page and 
many students forgot to turn the survey over.] The analysis below is therefore 
based on the sum of Items 1-8 only. The interest survey was only administered at 
two time points (pretest and posttest]. 

The Group x Time effect was significant (F(l, 110] = 3.98, p < .05], In the 
experimental group, the means increased by one point (from 22.25 to 23.25, p = 
.015, d = .23] but did not change in the control group (23.01 at pretest, 22.9 at 
posttest, p = .69], The increase in interest was driven by the first two items on the 
survey (learning more about where the Las Vegas Valley gets its water, p = .002, and 
learning more about home water conservation, p < .001], 

Student feedback survey 

For the experimental group, the mean score on the Likert-scale items (with some 
items reversed scored] was 4.19 out of 6 (SD = 1.1, n = 60], indicating that student 
perceptions of the game were generally positive. The open-ended items were 
scored for emerging themes. In regards to the item, "Please list at least two things 
you like about Losing the Lake," 48.3% of the participants referred to fun or 
interactive features of the game and 18.3% listed specific visual elements (e.g., 
graphs, pictures, and visuals of the lake levels lowering]. In addition, 46.7% referred 
to new information that they learned from playing the game, e.g., "new, economical 
ways to save water that I never heard of like the grey water". (Because participants 
were asked to list two ideas, the percentages sum to more than 100%.] 

DISCUSSION 

Although there is ample evidence that games promote interest and motivation, 
the evidence that games promote conceptual understanding is scant, with very few 
well-controlled studies with clear learning goals (Honey & Hilton, 2012], The fact 
that our study employed a randomized control group and demonstrated some 
degree of conceptual development is therefore notable, as summarized below. 

The Losing the Lake game resulted in increased student knowledge, as displayed 
by higher scores (compared to the control group] on both objective and constructed 
response items. According to criteria proposed by Cohen (1988], the effect sizes 
were medium (> 0.5] for the former, and large (> 0.8] for the latter. The increase 
was generally maintained on a delayed posttest, which is somewhat impressive 
given the short engagement time playing the game. (Although the retention 
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interval tested was only about 11 days, the results do indicate that much of the 
content learned became consolidated into students' long-term memories.] The 
increase was particularly prominent on items related to the concept of water flow 
(the "water we use" comes from Lake Mead and the Rocky Mountain snowpack and 
is returned to Lake Mead], supporting our claim that a local water-theme is a 
meaningful and productive concept to frame the learning environment. Although 
water is a "local topic," it is also a general one because many areas of the world are 
now experiencing drought. The Losing the Lake game can be useful educationally in 
various locales as a case study in the nature of drought, climate change effects, and 
water conservation practices. 

In relation to water conservation, the Losing the Lake game increased knowledge 
regarding conservation practices (e.g., the importance of covering swimming pools], 
but not on all productive conservation options; this may have been partially due to 
the fact that the game only provided informative feedback on the options that the 
user chose. Finally, there was increased knowledge on a couple items related to 
climate (e.g., how it differs from weather]. Prior research has shown that the 
misconception that climate and weather are the same thing is widespread and 
ingrained (Gowda, Fox, & Magelky, 1997; Lombardi & Sinatra, 2012], Weather is 
something that we experience daily, whereas the concept of climate is long-term and 
more abstract. The fact that playing the Losing the Lake game helped address this 
misconception is therefore notable. Emphasizing the concept explicitly and in large 
type ("Climate Change. 30 Years"] and emphasizing a long-term perspective (e.g., 
recession of glaciers over the last 100 years] may have contributed to the effect. 

Playing the game also resulted in some increase in student interest, particularly 
on topics related to water flow and conservation. This result also supports the 
conclusion that learning about "water impacts" can be an effective entree to learning 
about climate change. The game was designed to address household water 
conservation first and then to progress to issues of greater complexity and 
abstraction (community, region, and global climate]. Many of the learning and 
interest gains pertained to concrete issues within students' immediate realm of 
experience (e.g., the household] and this result is consistent with prior research and 
recommendations regarding teaching about local and personally relevant impacts of 
(and responses to] climate change (c.f. Anderson, 2012], Although the theme is of 
local relevance, climate-related drought is a problem in many parts of the world, and 
the Losing the Lake game can be used both as a case study and as a way of teaching 
about water conservation practices (and about climate change]. 

Conceptual development regarding water 

In playing Losing the Lake, students appeared to have some difficulty with items 
addressing complex relationships, for example how earlier Spring snowmelts would 
result eventually in reduced—not increased—lake water levels through, for 
example, increased evaporation. We suspect that while the game addressed these 
relationships, students need more supplementary instruction and time to process 
how the various knowledge pieces learned from playing the game fit into a coherent 
system. 

On the other hand, playing the game (even once] did appear to create some 
conceptual development. As shown in Table 10, there were significant gains on 
items pertaining to water flow, specifically where drinking water comes from (the 
Rocky Mountains snowpack] and where it goes when residents are done with it 
(back to the lake, unless recycled for outside use as grey water]. In a prior study 
that was used to inform our game design (Nussbaum, Sinatra, & Owens, 2012], we 
found that even college undergraduates had little idea as to the source of their 
drinking water. Water is essential for life, however, so it is understandable that 
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students who played the Losing the Lake game became curious about these issues. 
Furthermore, the game appeared to help users develop a basic understanding of 
water flow loops. Although such an understanding does not necessarily involve 
interaction effects or significant feedback effects, it does lay the foundation for 
developing more sophisticated concepts through additional instruction tied to the 
game. We argue that what is innovative about our game is not the technology per se 
but the conceptualization that it helps develop related to water use and 
sustainability. 

LIMITATIONS AND CONCLUSION 

This study had some limitations, particularly in regards to external validity. The 
game was evaluated in only two seventh-grade classrooms. The results could very 
well differ with students with different amounts and types of background 
knowledge, at different grade levels, or at different geographic locations. Future 
research should address these issues. Another limitation of the study is that 
students played the game at different points in the curriculum (at the beginning of a 
unit on climate in one school versus right after such a unit in the other school]. 
Although in both schools there was more growth on the objective test items and 
conservation item in the treatment groups than in the comparison groups, future 
research should address which sort of curriculum alignment is preferable. 

Another topic for future research is the utility of using the game to educate 
preservice or in-service science teachers about water resources and climate change. 
Many science teachers did not learn about these topics during their formal 
education (NRC, 2011] and instructional modules such as Losing the Lake could be 
especially valuable in enhancing their pedagogical content knowledge. In one study, 
Owens, Rehmat, and Nussbaum [2013] examined the use of Losing the Lake in a 
preservice elementary science course, and the teacher candidates found the game 
informative and engaging. We note that interesting and unusual curriculum 
materials usually are in short supply, and our game was eagerly embraced by the 
participating teachers in this study and the one reported on in the present article. 

In closing, we note that the National Research Council (NRC, 2011], in their 
comprehensive review of climate change education, identified a need for more 
cross-disciplinary efforts that combine the teaching of scientific content knowledge 
with behavioral change strategies. (These behavioral change strategies encompass 
the disciplines of psychology and economics.] Although we did not measure the 
effect of Losing the Lake on behavior, Losing the Lake does address knowledge of 
productive conservation strategies. Education strategies need to be action oriented, 
as the NRC concluded that most individuals desire guidance on what they can do 
personally and immediately to counter environmental threats. Enhancing 
individuals' self-efficacy promotes learning and motivation (Moser & Diller, 2007], 
Nonetheless, science instruction typically focuses on content knowledge, either 
directly or through inquiry, rather than behavior. The Losing the Lake game, with its 
continual focus on water conservation, is one attempt to address this need while 
simultaneously teaching about some of the causes and effects of climate change 
through a water-based theme. 
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APPENDIX 

Losing the Lake - pre/post assessment objective items 

True /False questions (7): 

Directions : You will be asked seven true /false questions to determine your 
knowledge about the greenhouse effect and climate change. Please choose the 
answer that best fits the question. 

1. The greenhouse effect and the ozone layer are basically the same. T or *F 

2. Climate change and changes in weather are the same. T or *F 

3. Over the past 100 years global temperatures have increased. *T or F 

4. Humans cannot survive on Earth without the greenhouse effect. *T or F 

5. Human activities produce greenhouse gases. *T or F 

6. Requiring everyone in the community to cover their swimming pool would 
save very little water. T or *F 

7. Resorts and casinos waste a large amount of water. T or *F 

Multiple-choice questions (13): 

Directions : You will be asked thirteen multiple-choice questions to determine 
your knowledge about water conservation, climate change, and their effect on Lake 
Mead. Please choose the answer that best fits the question. 

1. There are ways to conserve water at home. Some ways save more water than 

others. Which of the following is the best way to conserve water at home? 

a) Stop washing the family car at home. 

b) Replace your old washing machine with an energy and water-efficient one. 

c) Install low-flow faucets in your home. 

d) ^Replace lawns with desert landscape. 
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2. There are ways to conserve water within your community. Some ways save 
more water than others. Which of the following is the best way to conserve water 
within your community? 

a] Reduce watering golf courses in the area. 

b] Shut down any outside casino fountains. 

c] *Raise water prices. 

d] Require everyone in the community to cover their swimming pools. 

3. Which of the following options would save the most water? 

a] Planting trees. 

b] Not washing cars at home. 

c] *Usinggrey water systems. 

d] Taking shorter showers. 

4. What happens to the water that we are finished using in our homes? 

a] It is returned into the ground. 

b] It evaporates. 

c] *It is returned to Lake Mead. 

d] It flows through sewer systems. 

5. Grey water systems recycle water that comes from: 

a] rain. 

b] *bathing. 

c] manufacturing. 

d] toilets. 

6. What has the greatest effect on water levels in the Colorado River? 

a] *The amount of snow in the Rocky Mountains. 

b] The amount of snow in the Sierra Nevada Mountains. 

c] The amount of rain in the area. 

d] The amount of water flowing into the Colorado River from other rivers. 

7. Most of the water from Lake Mead is used for_. 

a] everyday things in homes. 

b] *growing crops. 

c] watering golf courses and lawns. 

d] drinking. 

8. If Lake Mead water levels decline below 1,000 feet above sea level, what will 
then happen? 

a] Emergency water allocations will go into effect. 

b] *We will no longer be able to pump water from the lake. 

c] Hoover Dam will no longer be able to generate electricity. 

d] Swimming and fishing will no longer be allowed at the lake. 

9. Having cities cut back 20% on their water use is estimated to delay Lake Mead 
water levels from declining below 1,000 feet (above sea levels] for how many 
years? 

a] 1 

b] 3 

c] *5 

d] 7 

e] 10 
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10. Which of the following determines how much water Nevada can use from 
Lake Mead? 

a] *The Colorado River Compact of 1922. 

b] The Southern Nevada Water Authority. 

c] The United States Bureau of Reclamation. 

d] Nevada State Legislature. 

11. If climate change causes an increase in average temperature of the area, what 
happens to the water in Lake Mead? 

a] Nothing because higher temperatures will not affect water levels. 

b] Nothing because the climate is not changing. 

c] *The water in Lake Mead will lower due to less snowpack and therefore, less 
water flowing into the Lake. 

d] The water in Lake Mead will rise due to the melting of the snowpack and 
therefore, more water flowing into the Lake. 

12. Which of the following is a greenhouse gas? 

a] Carbon monoxide. 

b] Oxygen. 

c] *Carbon dioxide. 

d] Hydrogen. 

13. Which of the following most contributes to an increase in greenhouse gases? 

a] *Car emissions. 

b] Ozone hole. 

c] Smog. 

d] Chopping down trees. 
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